Permutation Complexity Bound on Out-Sample Error

Assume a standard setting with data D = {(xi, yi)}i=1, where (xi, yi) are sampled iid from the joint distribution p(x, y) on Rd×{±1}. Let H = {h : R 7→ {±1}} be a learning model which produces a hypothesis g ∈ H when given D (we use g for the hypothesis returned by the learning algorithm and h for a generic hypothesis in H). We assume the 0-1 loss, so the in-sample error is ein(h) = 1 2n ∑n i=1(1 − yih(xi)). The out-sample error eout(h) = 1 2 E [(1 − yh(x))]; the expectation is over the joint distribution p(x, y). We wish to bound eout(g). To do so, we will bound |eout(h)− ein(h)| uniformly over H for all distributions p(x, y); however, the bound itself will depend on the data, and hence the distribution. The classic distribution independent bound is the VC-bound (Vapnik and Chervonenkis, 1971); the hope is that by taking into account the data one can get a tighter bound. The data dependent permutation complexity1 for H is defined by: